Field of Application
The field of application of the invention is data analysis especially as it applies to (so-called) “Big Data” (see sub-section 1 “Big Data and Big Data Analytics” below). The methods, systems and overall technology and knowhow needed to execute data analyses is referred to in the industry by the term data analytics. Data analytics is considered a key competency for modern firms [1]. Modern data analytics technology is ubiquitous (see sub-section 3 below “Specific examples of data analytics application areas”). Data analytics encompasses a multitude of processes, methods and functionality (see sub-section 2 below “Types of data analytics”).
Data analytics cannot be performed effectively by humans alone due to the complexity of the tasks, the susceptibility of the human mind to various cognitive biases, and the volume and complexity of the data itself. Data analytics is especially useful and challenging when dealing with hard data/data analysis problems (which are often described by the term “Big Data”/“Big Data Analytics” (see sub-section 1 “Big Data and Big Data Analytics”).
1. Big Data and Big Data Analytics
Big Data Analytics problems are often defined as the ones that involve Big Data Volume, Big Data Velocity, and/or Big Data Variation [2].                Big Data Volume may be due to large numbers of variables, or big numbers of observed instances (objects or units of analysis), or both.        Big Data Velocity may be due to the speed via which data is produced (e.g., real time imaging or sensor data, or online digital content), or the high speed of analysis (e.g., real-time threat detection in defense applications, online fraud detection, digital advertising routing, high frequency trading, etc.).        Big Data Variation refers to datasets and corresponding fields where the data elements, or units of observations can have large variability that makes analysis hard. For example, in medicine one variable (diagnosis) may take thousands of values that can further be organized in interrelated hierarchically organized disease types.        
According to another definition, the aspect of data analysis that characterizes Big Data Analytics problems is its overall difficulty relative to current state of the art analytic capabilities. A broader definition of Big Data Analytics problems is thus adopted by some (e.g., the National Institutes of Health (NIH)), to denote all analysis situations that press the boundaries or exceed the capabilities of the current state of the art in analytics systems and technology. According to this definition, “hard” analytics problems are de facto part of Big Data Analytics [3].
2. Types of Data Analysis
The main types of data analytics [4] are:                a. Classification for Diagnostic or Attribution Analysis: where a typically computer-implemented system produces a table of assignments of objects into predefined categories on the basis of object characteristics.                    Examples: medical diagnosis; email spam detection; separation of documents as responsive and unresponsive in litigation.                        b. Regression for Diagnostic Analysis: where a typically computer-implemented system produces a table of assignments of numerical values to objects on the basis of object characteristics.                    Examples: automated grading of essays; assignment of relevance scores to documents for information retrieval; assignment of probability of fraud to a pending credit card transaction.                        c. Classification for Predictive Modeling: where a typically computer-implemented system produces a table of assignments of objects into predefined categories on the basis of object characteristics and where values address future states (i.e., system predicts the future).                    Examples: expected medical outcome after hospitalization; classification of loan applications as risky or not with respect to possible future default; prediction of electoral results.                        d. Regression for Predictive Modeling: where a typically computer-implemented system produces a table of assignments of numerical values to objects on the basis of object characteristics and where values address future states (i.e., system predicts the future).                    Examples: predict stock prices at a future time; predict likelihood for rain tomorrow; predict likelihood for future default on a loan.                        e. Explanatory Analysis: where a typically computer-implemented system produces a table of effects of one or more factors on one or more attributes of interest; also producing a catalogue of patterns or rules of influences.                    Examples: analysis of the effects of sociodemographic features on medical service utilization, political party preferences or consumer behavior.                        f. Causal Analysis: where a typically computer-implemented system produces a table or graph of causes-effect relationships and corresponding strengths of causal influences describing thus how specific phenomena causally affect a system of interest.                    Example: causal graph models of how gene expression of thousands of genes interact and regulate development of disease or response to treatment; causal graph models of how socioeconomic factors and media exposure affect consumer propensity to buy certain products; systems that optimize the number of experiments needed to understand the causal structure of a system and manipulate it to desired states.                        g. Network Science Analysis: where a typically computer-implemented system produces a table or graph description of how entities in a mg system inter-relate and define higher level properties of the system.                    Example: network analysis of social networks that describes how persons interrelate and can detect who is married to whom; network analysis of airports that reveal how the airport system has points of vulnerability (i.e., hubs) that are responsible for the adaptive properties of the airport transportation system (e.g., ability to keep the system running by rerouting flights in case of an airport closure).                        h. Feature selection, dimensionality reduction and data compression: where a typically computer-implemented system selects and then eliminates all variables that are irrelevant or redundant to a classification/regression, or explanatory or causal modeling (feature selection) task; or where such as system reduces a large number of variables to a small number of transformed variables that are necessary and sufficient for classification/regression, or explanatory or causal modeling (dimensionality reduction or data compression).                    Example: in order to perform web classification into family-friendly ones or not, web site contents are first cleared of all words or content that is not necessary for the desired classification.                        i. Subtype and data structure discovery: where analysis seeks to organize objects into groups with similar characteristics or discover other structure in the data.                    Example: clustering of merchandize such that items grouped together are typically being bought together; grouping of customers into marketing segments with uniform buying behaviors.                        j. Feature construction: where a typically computer-implemented system pre-processes and transforms variables in ways that enable the other goals of analysis. Such pre-processing may be grouping, abstracting, existing features or constructing new features that represent higher order relationships, interactions etc.                    Example: when analyzing hospital data for predicting and explaining high-cost patients, co-morbidity variables are grouped in order to reduce the number of categories from thousands to a few dozen which then facilitates the main (predictive) analysis; in algorithmic trading, extracting trends out of individual time-stamped variables and replacing the original variables with trend information facilitates prediction of future stock prices.                        k. Data and analysis parallelization, chunking, and distribution: where a typically computer-implemented system performs a variety of analyses (e.g., predictive modeling, diagnosis, causal analysis) using federated databases, parallel computer systems, and modularizes analysis in small manageable pieces, and assembles results into a coherent analysis.                    Example: in a global analysis of human capital retention a world-wide conglomerate with 2,000 personnel databases in 50 countries across 1,000 subsidiaries, can obtain predictive models for retention applicable across the enterprise without having to create one big database for analysis.3. Specific Examples of Data Analytics Application Areas                        
The following Listing provides examples of some of the major fields of application for the invented system specifically, and Data Analytics more broadly [5]:                1. Credit risk/Creditworthiness prediction.        2. Credit card and general fraud detection.        3. Intention and threat detection.        4. Sentiment analysis.        5. Information retrieval filtering, ranking, and search.        6. Email spam detection.        7. Network intrusion detection.        8. Web site classification and filtering.        9. Matchmaking.        10. Predict success of movies.        11. Police and national security applications        12. Predict outcomes of elections.        13. Predict prices or trends of stock markets.        14. Recommend purchases.        15. Online advertising.        16. Human Capital/Resources: recruitment, retention, task selection, compensation.        17. Research and Development.        18. Financial Performance.        19. Product and Service Quality.        20. Client management (selection, loyalty, service).        21. Product and service pricing.        22. Evaluate and predict academic performance and impact.        23. Litigation: predictive coding, outcome/cost/duration prediction, bias of courts, voire dire.        24. Games (e.g., chess, backgammon, jeopardy).        25. Econometrics analysis.        26. University admissions modeling.        27. Mapping fields of activity.        28. Movie recommendations.        29. Analysis of promotion and tenure strategies.        30. Intension detection and lie detection based on fMRI readings.        31. Dynamic Control (e.g., autonomous systems such as vehicles, missiles; industrial robots; prosthetic limbs).        32. Supply chain management.        33. Optimizing medical outcomes, safety, patient experience, cost, profit margin in healthcare systems.        34. Molecular profiling and sequencing based diagnostics, prognostics, companion drugs and personalized medicine.        35. Medical diagnosis, prognosis and risk assessment.        36. Automated grading of essays.        37. Detection of plagiarism.        38. Weather and other physical phenomena forecasting.        
Discovery of causal networks in particular, is essential for understanding mechanisms of complex systems in various application domains. The inventive method uses local causal discovery methods for global causal network learning in a divide-and-conquer fashion.
The invention can be applied to practically any field where discovery of causal or predictive models is desired however because it relies on extremely broad distributional assumptions that are valid in numerous fields. Because the discovery of causal models facilitates feature selection, model conversion and explanation, inference and practically all aspects of data analytics, the invention is applicable and useful all the above mentioned types of data analysis and application areas.
Description of Related Art
A precursor to the main idea behind the local-to-global learning approach can be found in SCA [6], where a heuristic approximation of the local causes of every variable constraints the space of search of the standard Greedy search-and-score Bayesian method for global learning increasing thus computational efficiency. Other related methods include MMHC [7], Greedy Search, GES [8], OR [9], PC [10], and TPDA [11, 12]. Methods for inference of gene regulatory/transcriptional networks from high-dimensional gene expression data, the application domain of the present invention, are shown in FIG. 10.